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Abstract —  The  success  of  complex  systems  projects  is 
strongly  influenced  by  their  architecture.  A  key  role  of  a 
system  architect  is  to  decide  whether  and  how  to  integrate 
new  technologies  in  a  system  architecture.  Technology 
readiness  levels  (TRL)  scale  has  been  used  for  decades  to 
support  decision  making  regarding  the  technology 
infusion  in  complex  systems,  but  it  still  faces  challenges 
related  to  the  integration  of  technologies  to  a  system 
architecture.  Integration  Readiness  Levels  (IRL)  scale 
has  been  elaborated  in  the  last  decade  to  face  these 
challenges,  representing  the  integration  maturity  between 
the  technological  elements  of  a  system.  The  aim  of  this 
theoretical  article  is  to  perform  a  literature  review  on  IRL 
scale  evaluation  and  on  systems  architecture,  through 
bibliographic  research.  Results  show  the  review 
organized  in  five  topics  that  surrounds  the  research 
objective,  presenting  the  IRL  and  TRL  scales  evolution, 
comparing  their  evaluation  practices,  and  exploring  the 
architecture  complexity  of  systems.  Suggestions  for  future 
research  are  proposed  based  on  these  results. 

Keywords —  Integration  Readiness  Levels,  systems 
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I.  INTRODUCTION 

Systems  architecture  has  strong  influence  on  the  success 
or  failure  of  complex  systems  (Maier  &  Rechtin,  2000), 
and  one  of  the  key  roles  of  a  system  architect  is  to  decide 
whether  and  how  to  integrate  new  technologies  in  a 
system  architecture  (Crawley,  Cameron,  &  Selva,  2016). 
Complex  Products  and  Systems  (CoPS)  are  defined 
(Hobday,  1998)  as  high  cost  and  engineering  intensive 
products,  systems,  networks  and  buildings. 

The  Technology  Readiness  Levels  (TRL)  scale  was 
developed  to  support  decision  making  in  relation  to  the 
introduction  of  technologies  during  the  development  of 
complex  systems  (Mankins,  2009).  Although  this  scale 
has  been  used  for  decades,  it  does  not  reflect  well  the 
integration  of  technological  elements  into  the  system 
architecture,  and  its  application  has  other  challenges 
related  to  system  complexity,  project  planning, 
subjectivity  and  imprecision  of  the  scale  (Olechowski, 
Eppinger,  &  Joglekar,  2015). 


In  the  space  systems  industry,  the  current  global  scenario 
presents  notable  factors  such  as  intense  technological 
innovation,  growing  globalization,  entrepreneurship, 
proliferation  of  increasingly  smaller  satellites,  and 
product  modularization  (Futron,  2014).  Many  techniques 
used  for  space  systems  development  were  conceived  at 
the  time  of  the  space  race,  where  the  projects  had  large 
budget  and  greater  continuity  in  planning  (D.  Hastings, 
2004;  Ross,  Hastings,  Warmkessel,  &  Diller,  2004). 
Given  the  current  scenario  of  greater  technological, 
commercial,  political  and  application  uncertainties,  space- 
system  architectures  must  face  such  uncertainties  (D.  E. 
Hastings,  Weigel,  &  Walton,  2003),  and  technology 
readiness  assessment  methodologies  should  be  updated 
(Olechowski  et  al.,  2015). 

The  Integration  Readiness  Levels  (IRL)  scale  was 
proposed  to  represent  the  maturity  of  the  integration 
between  technological  elements  of  a  system  (Sauser, 
Verma,  Ramirez -Marquez,  &  Gove,  2006)  and  has  been 
evolving  over  the  last  decade. 

The  objective  of  this  research  is  to  perform  a  literature 
review  on  IRL  scale  evaluation  and  systems  architecture. 
The  literature  review  aims  to  compare  the  incipient  IRL 
evolution  and  evaluation  practices  to  the  more 
consolidated  TRL  literature,  and  to  explore  the  systems 
complexity  environment  where  both  scales  are  used,  by 
reviewing  concepts  related  to  systems  architecture, 
integration  and  their  representation. 

H.  METHODOLOGY 

Research  methodology  consisted  in  bibliographic  research 
with  qualitative  analysis,  comprising  five  topics.  The  first 
topic  presents  the  TRL  scale  fundamentals  and  current 
limitations.  In  the  second  topic,  the  IRL  scale  is  analyzed 
through  an  historical  perspective  and  according  to  topics 
of  interest  for  this  research.  The  third  topic  presents 
methodologies  and  best  practices  related  to  TRL 
assessment  process  and  the  equivalent  IRL  assessment 
process.  The  fourth  topic  presents  concepts  about  systems 
architecture  and  integration.  The  last  topic  shows  selected 
concepts  about  the  representation  of  dependencies  in 
complex  systems. 
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HL  RESULTS  AND  DISCUSSION 

3.1  TECHNOLOGY  READINESS  LEVELS 
According  to  Mankins  (2009),  in  the  1970s  the  National 
Aeronautics  and  Space  Administration  (NASA) 
introduced  the  concept  of  Technology  Readiness  Levels 
(TRL)  as  an  interdisciplinary  scale  to  allow  better 
assessment  and  communication  related  to  new 
technologies  development. 

The  main  objective  of  the  TRL  scale  is  to  assist  the 
decision  making  regarding  technology  infusion  in 
complex  systems  development.  When  a  technology  is  not 
mature  enough,  its  introduction  in  a  system  under 
development  may  lead  to  deviations  in  the  project 
schedule,  budget  and  performance  (GAO,  1999;  Mankins, 
2009;  Olechowski  et  al.,  2015). 

The  TRL  scale  was  modified  during  its  decades  of 
existence.  The  scale  was  originally  conceived  to  assist  the 
transition  from  technology  development  projects  to  space 
missions  development  (Sadin,  Povinelli,  &  Rosen,  1989). 
In  1995,  TRL  scale  was  strengthened  by  a  NASA 
publication  (Mankins,  1995)  which  detailed  each 
technology  readiness  level  definition  and  provided  TRL 
application  examples.  This  latest  version  of  the  TRL  scale 
considers  nine  discrete  levels  (1  to  9),  where  higher  TRL 
ratings  relate  to  more  mature  technologies. 

In  the  United  States,  Government  Accountability  Office 
(GAO)  in  1999  recommended  to  the  Department  of 
Defense  (DoD)  to  adopt  the  NASA  TRL  scale  or  similar 
scale  to  improve  the  research  and  development  results 
(GAO,  1999).  The  DoD  adopted  the  TRL  scale  with  some 
changes  to  the  original  version  (DoD,  2011).  Also  the 
Department  of  Energy  (DOE)  adopted  the  TRL  scale  with 
major  modifications  to  the  original  version  (DOE,  2015). 
In  the  2000s,  the  TRL  scale  began  to  be  used  in  space 
programs  from  other  regions  such  as  Europe  and  Japan 
(Mankins,  2009).  In  2013,  the  ISO  16290  standard  "Space 
systems  -  Definition  of  the  Technology  Readiness  Levels 
(TRLs)  and  their  criteria  of  assessment"  (ISO,  2013)  was 
published.  This  standard  was  proposed  by  European 
Cooperation  for  Space  Standardization  (ECSS,  2017a) 
and  discussed  at  international  level  by  the  International 
Organization  for  Standardization  (ISO)  committee 
members. 

According  to  ECSS  (2017a),  the  TRL  scale  proposed  in 
ISO  16290:2013  (ISO,  2013)  standard  presents  some 
differences  to  the  original  TRL  scale  (Mankins,  1995), 
which  are:  Level  5  is  a  new  intermediate  level  defined 
when  subscale  breadboards  are  used;  level  6  is  equivalent 
to  the  original  TRL  level  5;  level  7  is  equivalent  to  the 
original  TRL  level  6;  The  original  level  7  relates  to 
system  prototype  demonstration  in  space  environment  and 
is  not  defined  by  the  ISO  16290:2013  (ISO,  2013)  TRL 
scale. 
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According  to  Cornford  and  Sarsfield  (2004),  the  TRL 
scale  is  focused  on  a  particular  technology  evaluation,  and 
significant  integration  challenges  may  occur  when  a 
technology  is  included  in  a  space  system.  So  even  that  the 
technology  is  mature,  using  this  technology  in  new 
applications  may  be  challenging,  and  the  TRL  scale 
usually  does  not  represent  the  challenge  of  integrating  the 
technology  in  a  space  system. 

Olechowski,  Eppinger  and  Joglekar  (2015)  investigated 
the  use  of  the  TRL  scale  in  different  industrial  sectors 
through  interviews  and  analyzes  on  industry  standards 
and  organizational  guidelines.  These  authors  (Olechowski 
et  al.,  2015)  found  that  the  TRL  scale  is  widely  used  in 
different  complex  systems  industries  and  identified  fifteen 
challenges  to  improve  the  TRL  scale  utilization, 
categorized  in  three  topics:  system  complexity,  planning 
and  review,  and  assessment  validity. 

Subsequently,  Tomaschek,  Olechowski,  Eppinger  and 
Joglekar  (2016)  conducted  a  survey  with  TRL  scale 
practitioners  in  different  industries  worldwide  to  identify, 
among  the  fifteen  identified  challenges,  which  were  the 
most  priority  challenges.  The  survey  results  show  that  the 
four  highest  priority  challenges  are  related  to  the  systems 
complexity,  and  they  are:  representation  of  the  integration 
between  technologies,  interfaces  maturity,  modifications 
in  the  system  and  system  overall  maturity. 

3.2  INTEGRATION  AND  SYSTEM  READINESS 
LEVELS 

Research  initiated  at  the  Stevens  Institute  of  Technology, 
led  by  the  researcher  Brian  J.  Sauser,  proposed  two  new 
readiness  levels  scales  (Sauser  et  al.,  2006)  to 
complement  the  TRL  scale,  as  options  to  overcome  the 
TRL  scale  challenges  related  to  the  systems  complexity, 
the  same  challenges  identified  by  Tomaschek  et  al. 
(2016).  The  two  new  scales  proposed  were:  Integration 
Readiness  Levels  (IRL)  and  System  Readiness  Levels 
(SRL). 

The  research  about  the  integration  maturity  between 
components  in  a  system  is  also  justified  by  the  fact  that 
failures  of  many  space  systems  are  related  to  the 
integration  of  components  (Sauser,  Reilly,  &  Shenhar, 
2009). 

For  Sauser,  Ramirez-Marquez,  Henry  and  DiMarzio 
(2008),  technology  integration  is  part  of  the  systems 
engineering  effort  and  demands  a  quantitative  assessment 
tool  to  evaluate  the  risk  of  technology  integration  in  a 
complex  system. 

Sauser,  Gove,  Forbes  and  Ramirez-Marquez  (2010) 
consider  the  systems  integration  definition  proposed  by 
Buede  (2000),  as  the  aggregation  process  from 
components  that  need  to  be  aggregated  from  the  system 
configuration  items.  Sauser,  et  al.  (2010)  also  consider  the 
integration  process  as  the  upward  slope  in  a  "V"  model 
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commonly  used  in  systems  engineering.  Subsequently, 
the  scale  was  modified  to  also  represent  the  architecture 
definition  activities,  the  downward  slope  in  "V"  model  in 
systems  engineering. 

Sauser,  et  al.  (2010)  propose  that  the  IRL  scale  should  be 
able  to  be  applied  to  different  hierarchical  levels  from 
configuration  items  to  the  system  level,  and  that  the  scale 
should  represent  the  integration  in  sufficiently  general 
terms,  but  specific  enough  to  be  useful.  Sauser,  et  al. 
(2010)  suggested  that  scales  have  been  extensively  used 
to  support  the  integration  between  components  in  the 
computer  industry,  but  scales  to  support  more  general 
systems  integration  are  less  developed. 

The  first  version  of  the  IRL  scale  (Sauser  et  al.,  2006)  was 
designed  using  the  International  Standards  Organization's 
Open  Systems  Interconnect  (ISO  /  OSI)  scale,  used  in 
computer  networks,  which  represents  data  integration 
levels  in  a  particular  interface  between  one  or  more 
systems.  The  first  version  of  the  IRL  scale  had  seven 
readiness  levels,  based  on  the  ISO  /  OSI  scale. 

Sauser,  et  al.  (2010)  included  two  new  levels  to  the  IRL 
scale,  when  compared  to  the  original  IRL  version  (Sauser 
et  al.,  2006).  The  two  new  levels  were:  Level  8  related  to 
qualification  through  testing  and  demonstration  and  Level 
9  related  to  the  successful  operation  in  a  mission. 

Further,  the  IRL  scale  was  modified  to  better  reflect  the 
systems  development  process  and  to  be  more  consistent 
with  the  fundamentals  of  TRL  scale  (Austin  &  York, 
2015,2016). 

The  IRL  scale  is  commonly  assessed  using  a  Design 
Structure  Matrix  (DSM)  to  represent  the  integration 
between  the  system  components  (Olechowski  et  al., 
2015). 

The  System  Readiness  Levels  scale,  or  SRL,  was 
proposed  to  quantify  the  readiness  level  of  a  component 
in  relation  to  the  other  components  that  constitute  a 
system,  and  indicate  how  much  the  whole  system  is 
integrated  (Sauser,  Ramirez-marquez,  &  Tan,  2008). 
Equation  1  (Austin  &  York,  2015)  shows  the  composite 
SRL  calculation  for  a  system  with  'N'  components,  where 
the  matrix  [SRL]nxi  is  obtained  by  multiplying  the  matrix 
[IRL]nxn,  which  represents  the  integration  readiness 
levels  between  the  'N'  components,  and  the  matrix 
[TRL]nxi,  which  represents  the  technology  readiness 
levels  for  each  of  the  'N'  components.  The  SRL  of  the 
overall  system  can  be  obtained  by  the  normalized  average 
of  the  elements  of  the  matrix  [SRLJnxi- 

[SRL]  Nxl  =  [IRL]nxn  X  [TRL]  Nxl  (1) 

The  SRL  scale  can  be  transformed  in  a  scale  of  discrete 
numbers  comprising  certain  calculated  SRL  intervals 
(Austin  &  York,  2015,  2016). 
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Other  applications  proposed  for  the  IRL  and  SRL  scales, 
together  with  TRL  scale,  are:  use  the  readiness  levels  as  a 
baseline  coupled  to  a  earned  value  management  system 
for  project  management  (Magnaye,  Sauser,  Patanakul, 
Nowicki,  &  Randall,  2014);  system  development 
planning  and  system  development  costs  minimization 
(Magnaye,  Sauser,  &  Ramirez-Marquez,  2010);  and  the 
association  with  effectiveness  metrics  for  systems  design, 
such  as  the  Equivalent  Mass  System  (Sauser  &  Magnaye, 
2010). 

The  IRL  scale  is  being  used  in  the  aerospace  industry 
(Sauser,  Long,  Forbes,  &  McGrory,  2009),  and 
customized  to  reflect  the  system  integration  process  for 
oil  and  gas  exploration  (Knaggs  et  al.,  2015,  2017, 
Yasseri,  2013,  2016). 

Other  papers  communicate  the  experience  evaluation 
about  using  the  IRL  scale  (Atwater  &  Uzdzinski,  2014; 
Baiocco  et  al.,  2015;  Knaggs  et  al.,  2017;  Lemos  & 
Chagas  Ir.,  2016;  London,  Holzer,  Eveleigh,  &  Sarkani, 
2014;  Mantere,  2014;  Mantere  &  Pirinen,  2014; 
Mapamba,  Conradie,  &  Fick,  2016;  Mcconkie,  Mazzuchi, 
Sarkani,  &  Marchette,  2013;  Pirinen,  2014;  Sivlen  & 
Pirinen,  2014). 

Kujawski  (2013)  criticized  the  SRL  scale,  with  the 
argument  that  the  scale  is  a  product  between  two  ordinal 
numbers,  which  represent  the  TRL  and  IRL  scales,  and  its 
results  should  be  analyzed  with  caution.  Jimenez  and 
Mavris  (Jimenez  &  Mavris,  2014)  criticized  the  IRL 
scale,  at  the  time  that  the  IRL  scale  was  based  on  the  ISO 
/  OSI  data  integration  scale,  proposing  that  the  scale  was 
very  specific  to  the  data  management  effort,  and 
suggested  to  use  only  the  TRL  scale.  However,  the  IRL 
scale  has  evolved  to  address  this  criticism. 

The  International  Systems  Readiness  Assessment 
Community  of  Interest  (ISRACOI,  2018)  is  a  worldwide 
collaborative  community  of  researchers,  practitioners  and 
stakeholders  interested  in  system  readiness  metrics  such 
as  TRL,  IRL  and  SRL.  New  researches  and  white  papers 
are  published  in  the  ISRACOI  website  (2018). 

3.3  TECHNOLOGY  READINESS  ASSESSMENT 
According  to  Mankins  (2009),  key  factors  for  effective 
use  of  the  TRL  scale  are:  to  perform  objective  and  well 
documented  assessments  for  the  readiness  and  risks  about 
the  technology  under  evaluation;  and  perform  the 
assessments  at  critical  decision  making  milestones  in  the 
complex  system  development  project.  Mankins  (2009) 
proposed  the  concept  of  Technology  Readiness 
Assessment  (TRA)  as  a  methodology  used  to  conduct  the 
TRL  scale  evaluation  process.  For  Mankins  (2009),  a 
rigorous  TRA  should  include  clear  evidences  that  the 
declared  TRL  was  achieved  -  such  as  photos  of  a 
breadboard  in  the  laboratory,  quantitative  data  verification 
tests,  among  other  evidences. 
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The  main  methodologies  used  to  perform  a  TRA  have 
been  qualitative  analysis  from  experts  to  establish  the 
adequate  TRL  level,  and  analysis  supported  by  structured 
interviews  and  quantitative  methods  (Hueter  &  Tyson, 
2010).  These  quantitative  methods  may  be  supported  by 
probability  and  statistics  analyses  (Ristinen,  2010). 
Bayesian  networks  may  also  be  applied  to  calculate  the 
most  adequate  TRL  level  (Austin  et  al.,  2017). 

Air  Force  Research  Laboratory  (AFRL)  developed  an 
automated  tool  (Nolte,  Kennedy,  &  Dziegiel,  2003)  to 
structure  the  TRL  assessment  interview  and  quantify  the 
TRL  level  more  appropriate  to  the  responses.  The 
questionnaire  presents  questions  regarding  the  level  of 
knowledge  about  the  technology  and  its  potential 
customers,  about  the  technology  development 
documentation,  future  aspects  of  integration,  modeling 
and  simulation,  verification  and  system  environment 
where  the  technology  might  be  infused.  According  to 
these  authors  (Nolte  et  al.,  2003),  the  main  application 
envisioned  for  the  TRL  scale  in  this  case  was  to  guide  the 
transition  from  technology  development  programs  to  the 
technology  use  in  operational  systems. 

Similarly,  the  Jet  Propulsion  Laboratory  (JPL)  developed 
a  TRA  process  (Frerking  &  Beauchamp,  2016)  based  on 
NASA's  recommendations.  This  process  contains  a 
questionnaire  based  on  a  previous  questionnaire  proposed 
by  Bilbro  (2009). 

Cornford  and  Sarsfield  (2004)  argue  that:  the  TRL 
assessment  techniques  at  that  moment  were  qualitative; 
and  that  the  importance  of  the  language  and  culture 
involved  in  the  technology  transfer  process  between 
laboratories  and  their  integration  in  space  systems  was 
generally  underestimated.  In  this  way,  the  TRL 
assessments  were  subjective  due  to  factors  such  as:  the 
team  carrying  out  the  assessment  was  usually  the  same 
development  team  and  not  a  third  party  with  more 
impartial  view;  the  original  settings  of  the  TRL  scale 
could  be  interpreted  in  different  ways;  even  though 
quantitative  methods  were  used,  the  database  was  still 
subjective  due  to  limitations  in  the  TRL  scale  definitions 
objectivity. 

A  list  of  supporting  information  for  each  TRL  level  was 
developed  by  the  DoD  (2011)  in  order  to  make  more 
objective  assessments,  containing  information  related  to 
systems  engineering,  verification,  technical  requirements 
and  circumstances  relevant  to  each  TRL  level. 

The  ISO  16290:2013  standard  (ISO,  2013)  presents  a  list 
with  the  work  performed  and  documented  as  evidences 
required  to  achieve  each  level  TRL.  This  list  of  evidences 
presents  predominantly  elements  of  space  systems 
verification  discipline,  accompanied  by  their  respective 
performance  requirements  and  technology  definition 
documents  (ECSS,  2017a). 
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GAO  (2016)  published  a  preliminary  report  in  order  to 
establish  a  methodology  based  on  best  practices  that 
could  be  used  by  the  USA  federal  government  to  perform 
technology  readiness  assessments  (TRA),  aiming  mainly 
to  support  decision  making  on  programs  and  projects 
which  involve  large  commitments  of  financial  resources. 
Some  of  the  TRA  best  practices  described  in  this 
document  (GAO,  2016)  are: 

•  The  responsible  for  the  TRA  should  understand 
which  evidences  would  be  needed  for  the  TRL  scale 
and  understand  the  operational  environment  in 
which  the  technology  should  operate,  depending  on 
whether  the  assessment  is  performed  to  meet 
government  agencies  requirements  or  it  is  performed 
as  an  internal  exercise  to  monitor  the  technology 
readiness; 

•  Reliable  assessments  are  supported  by  artifacts  and 
clear  information,  such  as  requirements  documents, 
analyzes,  test  reports  and  environmental  testing 
considerations; 

•  Supporting  information  and  evidence  needed  for 
each  TRL  level  are  best  practices.  They  are  not 
exhaustive  and  may  vary  according  to  the 
technology  or  application,  so  it  is  necessary  to  adapt 
these  definitions  to  better  reflect  the  technology  and 
its  application; 

•  The  quality  of  a  TRA  depends  on  the  accuracy  and 
relevance  of  the  artifacts,  test  data,  analysis  reports, 
and  other  supporting  information.  This  information 
can  be  dependent  on  other  technologies  or  activities 
that  are  outside  the  assessment  scope,  but  may  need 
to  be  included  to  better  assess  the  technology 
readiness.  Changes  or  refinements  in  requirements, 
technology  parameters  or  other  factors  may  affect 
the  TRA,  in  which  case  the  TRA  should  be  updated. 

ECSS  published  the  handbook  ECSS-E-HB-1 1A 
"Technology  readiness  level  (TRL)  guidelines"  (ECSS, 
2017a)  as  a  guide  to  the  TRL  scale  application  in  space 
missions  and  programs,  offering  guidelines  for  the 
interpretation  of  the  TRL  scale  and  best  practices  related 
to  the  technology  readiness  assessment  process.  ECSS 
adopted  the  ISO  16290:  2013  (ISO,  2013)  standard  for  the 
TRL  scale  definition. 

Regarding  the  IRL  scale,  Sauser,  et  al.  (2010)  established 
decision  criteria  to  support  the  assessment  for  each  IRL 
level.  The  criteria  were  based  on  two  sources.  The  first 
source  was  the  evaluation  of  standards,  researches  and 
other  documents  related  to  systems  engineering  and 
acquisition  processes  (such  as  DoD  5000.02,  INCOSE 
Systems  Engineering  Handbook,  IEEE  15288,  NASA 
Systems  Engineering  Handbook).  The  second  source  was 
based  on  discussions  and  interviews  with  experts  in  the 
areas  of  systems  engineering,  project  management  and 
procurement  management,  to  assess  what  would  be  the 
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most  important  decision  criteria  for  each  IRL  level.  As 
integration  is  a  complex  topic,  Sauser,  et  al.  (2010) 
recommended  that  future  researches  could  continue  to 
review  and  modify  the  decision  criteria  list  and  their 
relative  importance. 

Austin  and  York  (2015,  2016)  presented  the  latest  version 
of  the  IRL  scale.  A  column  in  the  IRL  scale  presents  the 
required  evidences  for  each  level,  incorporating  the  most 
relevant  criteria  identified  by  Sauser,  et  al.  (2010)  and 
including  data  integration  testing  particularities. 

The  list  of  required  evidences  for  the  IRL  scale  presents  a 
potential  improvement  opportunity  (Jesus  &  Chagas  Jr, 
2017)  for  its  definition,  when  compared  to  how  the  list  of 
required  evidences  for  the  TRL  scale  is  structured  in  the 
ISO  16290:2013  (ISO,  2013)  standard. 

The  integrated  process  for  assessing  the  TRL,  IRL  and 
SRL  scales  is  defined  as  the  System  Readiness 
Assessment  (SRA)  (Austin  &  York,  2015,  2016).  A 
system  mapping  provides  an  understanding  of  the 
relationships  between  the  different  architecture  layers. 
The  highest  hierarchical  level  of  this  mapping  is  based  on 
operational  requirements  and  activities.  Then  the 
functions  supporting  these  operational  activities  are 
mapped.  After  that,  the  system  components  that  perform 
these  functions  are  identified.  In  turn,  the  components  are 
composed  by  technologies.  Connection  diagrams  between 
the  components  help  to  understand  the  system 
architecture  and  integration.  Fig.  1  illustrates  an  example 
of  SRA  application  (Austin  &  York,  2015). 


Fig.l:  Example  of  System  Readiness  Assessment 
application. 
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Source:  Austin  and  York  (2015),  which  is  published 
under  a  CC  BY-NC-ND  license  (Creative  Commons, 
2018). 

3.4  COMPLEX  SYSTEMS  ARCHITECTURE  AND 
INTEGRATION 

Hobday  (1998)  defined  Complex  Products  and  Systems 
(CoPS)  as  high  cost  and  engineering  intensive  products, 
systems,  networks  and  buildings.  CoPS  tend  to  be 
manufactured  in  single  projects  or  in  small  batches  and 
the  production  emphasis  tends  to  be  on  design,  project 
management,  systems  engineering  and  systems 
integration.  Examples  of  CoPS  include  satellites, 
telecommunications  networks,  flight  simulators,  aircraft 
engines,  avionics  systems,  train  engines,  air  traffic  control 
units,  electrical  network  systems,  offshore  oil  equipment, 
intelligent  buildings  and  telephone  network  equipment. 
Due  to  its  high  cost  and  customization  features,  the 
dynamics  of  innovation  and  the  nature  of  the  industrial 
coordination  are  different  in  relation  to  other  types  of 
products,  especially  the  low  cost,  mass-produced,  and 
based  on  standard  components. 

Hobday,  Davies  and  Prencipe  (2005)  suggested  that 
systems  integration  became  an  essential  capability  for 
modern  corporations.  Many  major  global  companies  are 
developing  a  new  industrial  organization  model  based  on 
systems  integration.  Instead  of  performing  all  the 
productive  tasks  in-house,  companies  are  building 
capabilities  to  design  and  integrate  systems,  while 
managing  networks  of  component  and  subsystem 
suppliers.  In  this  sense,  systems  engineering  and  project 
management  disciplines  are  needed  to  coordinate  the 
technical  and  organizational  effort  required  for  systems 
integration  (Eisner,  2008). 

Some  applications  of  readiness  scales  for  system 
integration  are:  analyze  the  depth  of  systems  integrators 
technology  base  (Chagas  Jr.,  Leite,  &  Jesus,  2017), 
categorize  (Lemos,  2016;  Shenhar  et  al.,  2005)  and 
support  (Jesus  &  Chagas  Jr.,  2016)  CoPS  development 
projects,  and  apply  to  a  technology  vigilance  system 
(Andrade,  Silva,  Chagas  Jr.,  Rosa,  &  Chimendes,  2017). 
Eisner  (2005)  identified  in  the  literature  factors  that 
contribute  to  greater  complexity  of  systems,  which  were: 
size,  number  of  functionalities,  parallel  versus  serial 
operation,  number  of  operating  modes,  duty  cycle 
(dynamic  versus  static),  real-time  operations,  very  high 
performance  level,  number  of  interfaces,  different  types 
of  interfaces,  degree  of  integration,  non-linear  behavior 
and  human-machine  interaction.  Regarding  the  different 
types  of  interfaces,  many  systems  have  a  simple 
mechanical  and  electrical  interface,  as  connecting  stereo 
components  and  connecting  a  cable  to  a  DVD  player, 
VCR  or  TV  set,  but  if  we  add  for  example  thermal. 
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environmental  interface  requirements,  data  structure  and 
protocols,  the  system  becomes  more  complex. 

Blanchard  and  Fabrycky  (2006)  proposed  the  following 
main  system  life  cycle  phases:  Conceptual  Design, 
Preliminary  Design,  Detail  Design  and  Development, 
Production/Construction,  Operational  use  and  system 
support.  Retirement.  The  activities  of  testing,  evaluation 
and  validation  of  the  system  are  progressively  carried  out 
throughout  its  development. 

According  to  Blanchard  and  Fabrycky  (2006),  models  are 
created  to  represent  a  system  under  study  and  can  be 
classified  as  physical,  analogue,  schematic,  and 
mathematical  types.  Physical  models  look  like  what  they 
represent,  analogue  models  behave  like  the  system, 
schematic  models  describe  graphically  a  process  or 
situation,  and  mathematical  models  represent 
symbolically  the  principles  of  a  situation  under  study. 
During  the  early  phases  of  detail  design,  breadboards, 
bench-test  models,  engineering  models,  engineering 
software,  and  service  test  models  are  built  aiming  to 
verify  specific  performance  or  physical  design 
characteristics.  Formal  tests  and  demonstrations  are 
carried  out  during  the  latter  activities  of  the  detailed 
design  phase  when  pre-production  prototype  equipment, 
software,  and  similar  formal  procedures  are  available. 
Also,  according  to  Blanchard  and  Fabrycky  (2006)  the 
basic  architecture  of  the  system  is  established  with  the 
definition  of  system  operational  requirements,  the  concept 
of  maintenance  and  support  and  the  identification  and 
prioritization  of  technical  performance  measures  (TPMs). 
A  system  architecture  represents  the  system  high-level 
design  or  configuration,  its  operational  interfaces, 
anticipated  usage  profiles  (mission  scenarios)  and  the 
environment  in  which  it  must  operate,  describing  how 
these  various  requirements  should  interact  for  the  system. 
Next,  the  functional  architecture  describes  the  system  in 
functional  terms.  From  this  analysis,  through  the 
requirements  allocation  process  and  the  definition  of  the 
various  resource  requirements  necessary  for  the  system  to 
reach  its  mission,  the  physical  architecture  is  defined. 
Maier  and  Rechtin  (2000)  proposed  that  the  progressive 
refinement  of  the  design  is  one  of  the  most  basic  patterns 
for  the  engineering  practice.  The  process  of  systems 
architecting  is  performed  through  the  progression,  or 
gradual  reduction  of  the  abstraction,  modeling,  evaluation 
criteria,  heuristics  and  purposes,  from  the  initial  ideas  up 
to  reach  the  most  formal  and  detailed  processes  in 
different  fields  of  engineering.  Thus,  the  evolution  and 
development  of  models  are  threated  as  the  core  process  of 
systems  architecting.  The  models  represent  and  control 
the  system  specification,  its  design  and  its  production 
plan.  Even  after  the  system  delivery,  the  modeling  will  be 
the  mechanism  to  evaluate  the  system  behavior  and  plan 
its  evolution. 
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Rechtin  (2000)  proposed  that  decisions  related  to  the 
evolution  or  creation  of  new  system  architectures 
influence  directly  the  competitiveness  of  organizations  to 
meet  the  demands  of  their  customers,  and  therefore 
influence  directly  the  success  of  these  organizations. 
Regarding  the  product  development  process,  Ulrich  and 
Eppinger  (2012)  proposed  that  product  architecture  is  the 
allocation  of  functional  elements  to  physical  elements  of 
the  product.  The  purpose  of  the  product  architecture  is  to 
define  the  basic  building  blocks  of  the  system  physical 
elements  in  terms  of  what  they  do  and  what  are  their 
interfaces  with  the  rest  of  the  product.  After  completing 
the  architecture  definition,  it  is  possible  to  perform  the 
detailed  design  and  testing  of  these  building  blocks, 
allocate  them  to  teams  or  suppliers,  so  that  the 
development  of  different  parts  of  the  product  could  be 
done  simultaneously.  Decisions  on  the  product 
architecture  and  modularity  influence  directly  important 
aspects  of  the  organization  success,  such  as  future 
changes  in  the  product  range,  components 
standardization,  product  performance,  manufacturing,  and 
product  development  management  (Ulrich,  1995;  Ulrich 
&  Eppinger,  2012). 

Crawley,  Cameron  and  Selva  (2016)  proposed  that  a 
simple  definition  for  system  architecture  is  the  abstract 
description  of  the  system  entities  and  the  relationships 
between  them,  and  proposed  that  in  systems  engineering 
the  architecture  can  be  represented  as  a  set  of  decisions. 
These  authors  (Crawley  et  ah,  2016)  proposed  that  system 
architecting  is  a  composition  of  science  and  art,  with  the 
rationalization  of  decisions  through  the  formulation  of 
how  these  decisions  can  impact  the  system  performance. 
These  authors  (Crawley  et  ah,  2016)  also  suggested  that 
the  system  architecture  can  be  used  to  map  and  analyze  an 
existing  system,  in  a  reverse  engineering  method,  or  be 
used  in  the  synthesis  of  a  new  system,  in  a  direct 
engineering  method. 

Crawley,  Cameron  and  Selva  (2016)  considered  that  a 
new  technology  is  often  at  the  heart  of  a  new  product,  and 
a  change  in  the  technology  is  often  a  major  motivation  for 
a  new  architecture  development.  So,  according  to  these 
authors  (Crawley  et  ah,  2016),  one  of  the  key  roles  of  the 
system  architect  is  to  decide  whether  and  how  to  infuse  a 
new  technology  into  a  system  architecture.  This  infusion 
would  require  deep  knowledge  of  the  available 
technology  and  its  maturity,  the  process  to  integrate  the 
technology  into  the  system,  and  the  value  that  this 
integration  would  create.  Still  according  to  these  authors 
(Crawley  et  ah,  2016),  the  TRL  scale  is  useful  to  support 
system  architects  to  take  these  decisions. 

Ross,  et  ah  (2004)  and  Hastings  (2004)  suggested  that 
many  techniques  to  support  space  systems  development 
were  conceived  during  the  space  race,  where  the  projects 
had  large  budgets  and  great  planning  continuity. 
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Nowadays,  in  addition  to  face  the  technical  challenges  in 
building  such  complex  systems,  engineers  must  also  deal 
with  changes  in  the  political  and  economic  context  that 
influence  the  design  and  development  of  space  systems. 
Hastings,  et  al.  (2003)  proposed  that  decisions  in  system 
architectures  should  help  to  address  these  uncertainties, 
with  a  focus  on  strategies  such  as  flexibility  and 
robustness  that  can  lead  designers  to  different 
optimization  solutions  to  meet  specifications  or  other 
specific  criteria. 

Crawley,  et  al.  (2004)  suggested  that  system  architectures 
are  not  static,  but  they  evolve  for  long  periods  as 
technologies  mature.  They  also  evolve  during  the  natural 
process  of  designing  a  system.  These  evolutionary 
patterns  are  useful  for  understanding  the  importance  of 
the  representation  and  the  decisions  involved  in  a  system 
architecture. 

In  line  with  the  previously  described  context  that  systems 
and  their  architectures  face,  Olechowski,  Eppinger  and 
Joglekar  (2015)  proposed  that  is  important  to  update  the 
TRL  scale  application  methods.  The  argument  is  that 
since  the  current  context  of  growing  systems  complexity, 
greater  dynamics  of  innovation,  the  current  use  of  TRL  in 
decision-making  and  the  current  use  in  different 
organizational  processes,  are  significantly  different  from 
the  context  experienced  by  NASA  in  the  1970s  when  the 
original  TRL  scale  was  created. 

Regarding  the  definition  for  systems  integration,  the 
systems  engineering  literature  and  standards  propose  a 
common  notion  for  systems  integration  as  the  process  of 
assembling  and  integrating  elements  of  smaller 
hierarchical  levels,  successively  into  larger  hierarchical 
levels,  until  the  system  and  its  desired  functionalities  are 
realized  (Buede,  2000;  DoD,  2017;  ECSS,  2017b;  IEEE, 
IEC,  &  ISO,  2007;  INCOSE,  2006;  Kossiakoff,  Sweet, 
Seymour,  &  Biemer,  2011;  NASA,  2017;  Peterson  & 
Rodberg,  2005).  The  ISO  /  IEC  26702-2007  IEEE  1220- 
2005  “Standard  for  Systems  Engineering  -  Application 
and  Management  of  the  Systems  Engineering  Process” 
(IEEE,  IEC  &  ISO,  2007)  added  that  to  perform  the 
integration  of  elements  into  a  system,  the  design  and 
interface  logic  of  these  elements  must  be  met. 

For  Sage  and  Lynch  (1998),  the  integration  of  technical 
parameters  and  interface  compatibility  are  generally 
assumed  as  a  technical  effort  carried  out  during  the  latter 
parts  of  the  system  development  lifecycle.  However,  if 
this  effort  is  not  planned  properly  in  the  definition  phases 
and  in  the  early  parts  of  the  development  phase, 
integration  is  likely  to  be  difficult  at  the  end  of  the 
development  phase.  To  perform  concurrent  engineering 
and  simultaneously  develop  different  parts  of  the  system, 
initial  efforts  must  be  made  in  system  architecture  and 
system  partitioning,  and  then  an  explicit  system 
integration  stage  is  required. 
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In  this  way.  Sage  and  Lynch  (1998)  suggested  that  if  we 
consider  that  the  activities  of  analysis,  definition,  design, 
requirements  and  control  of  interfaces  are  all  integration 
efforts,  then  the  system  integration  occurs  at  almost  every 
stage  of  the  systems  development  lifecycle,  not  just  in  its 
later  stages. 

In  his  proposal  for  a  general  system  integration  theory, 
Langford  (2013)  proposed  that  integration  is  the  approach 
of  building  or  creating  a  whole  from  parts,  and  is  more 
than  simply  combining  or  assembling  these  parts.  Many 
system  integration  efforts  undergo  changing  requirements 
for  many  reasons,  and  while  there  are  numerous  strategies 
for  solving  these  system  integration  problems,  it  is  time 
consuming  to  plan  and  integrate  any  part  of  the  system  as 
part-by-part,  because  the  problems  persist  (Ramamoorthy, 
Chandra,  Kim,  Shim,  &  Vij,  1992).  The  integration 
concept  proposed  by  Langford  (2013)  expresses 
integration  of  artifacts  as  "part-to-all-expected"  rather 
than  "part-to-part".  For  example,  to  integrate  parts  A  and 
B  to  reach  system  C:  Plan  and  integrate  Part  A  in  the  way 
that  system  C  should  behave,  and  then  Part  B  in  the  way 
that  system  C  should  behave,  is  more  effective  than 
integrating  Part  A  to  Part  B  to  reach  system  C.  If  Part  A  is 
not  available.  Part  B  can  still  be  integrated  with  the 
behaviors  of  C  to  show  how  Part  A  would  (and  should) 
behave.  According  to  this  author  (Langford,  2013),  the 
application  of  this  "part-to-all-expected"  system 
integration  concept  illustrates  the  power  of  thought  and 
theoretical  planning,  that  is,  most  situations  that  need  to 
be  addressed  during  integration  planning  could  be 
threated  regardless  of  the  individual  parts  situation. 

Zandi  (1986)  suggested  that  science  and  engineering 
should  make  use  of  systemic  thinking,  considering  that  a 
system  is  more  than  the  sum  of  its  parts,  possessing 
emergent  properties.  Crawley,  Cameron,  and  Selva  (2016) 
proposed  that  emergence  refers  to  what  appears, 
materializes,  or  surfaces  when  a  system  operates,  or  in 
other  words  the  functions  that  emerge  when  a  system 
operates.  Emerging  functions  can  be  classified  as  desired 
or  undesirable,  and  anticipated  or  unanticipated  (Crawley 
et  al.,  2016).  Sillitto  (2005)  proposed  that  systems 
engineering  could  be  defined  as  the  management  of 
emerging  properties,  such  as  the  importance  of  emerging 
properties  in  the  management  of  a  system. 

3.5  REPRESENTATION  OF  DEPENDENCIES  IN 
COMPLEX  SYSTEMS 

According  to  Browning  (2016),  the  Design  Structure 
Matrix  (DSM),  also  called  the  dependency  structure 
matrix,  became  a  modeling  framework  widely  used  in 
many  areas  of  research  and  practice.  DSM  has  advantages 
of  simplicity  and  conciseness  in  its  representation  and, 
supported  by  appropriate  analysis,  may  also  highlight 
important  patterns  in  system  architectures  (design 
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structures)  such  as  modules  and  clusters  (Eppinger  & 
Browning,  2012).  DSM  is  a  square  matrix  where  diagonal 
cells  normally  represent  the  elements  of  the  system  and 
off-diagonal  cells  represent  relations  (such  as 
dependencies,  interfaces,  interactions,  etc.)  between  the 
elements. 

Recent  publications  on  the  IRL  scale  use  a  DSM  matrix  to 
map  the  system  architecture  and  perform  the  IRL 
evaluation  (Olechowski  et  al.,  2015). 

The  N-Squared  (N2)  diagram  is  also  an  widely  used  tool 
(Crawley  et  al.,  2016;  Lalli,  Kastner,  &  Hartt,  1997; 
Larson,  Kirkpatrick,  Sellers,  Thomas,  &  Verma,  2009; 
NASA,  2017).  The  diagram,  in  a  matrix  form,  is  used  to 
represent  the  interfaces  of  a  system.  The  components  or 
functions  of  the  system  are  placed  diagonally,  while  the 
other  cells  in  the  NxN  matrix  represent  the  inputs  and 
outputs  of  the  interfaces,  the  outputs  being  represented  in 
rows  and  the  entries  in  columns.  Alternatively,  the 
interfaces  may  be  represented  without  polarization,  that 
is,  without  distinction  between  inputs  and  outputs.  The 
N2  diagram  may  be  applied  at  successively  lower 
hierarchical  system  levels,  and  may  also  represent 
external  interfaces  to  the  system.  The  N2  diagram 
application  is  similar  to  the  DSM  (Eppinger  &  Browning, 
2012). 

IV.  CONCLUSION 

This  paper  presented  a  literature  review  on  IRL  scale 
evaluation  and  systems  architecture,  covering  five  topics 
that  surrounded  the  main  topics  of  interest. 

The  fundamentals  of  the  TRL  scale  and  its  assessment 
methods  were  analyzed  in  order  to  understand  the  IRL 
origin  and  be  able  to  compare  both  scales  literatures. 
Systems  architecture  and  systems  integration  concepts 
were  presented  in  order  to  highlight  the  complexity  of 
systems  and  explore  the  context  where  these  scales  are 
used.  Selected  concepts  about  the  representation  of 
dependencies  in  complex  systems  are  shown  in  order  to 
identify  methodologies  being  practiced  in  the  literature. 
Through  the  literature  review  analysis,  it  is  possible  to 
suggest  future  researches  about  the  IRL  scale,  such  as; 
Evaluate  the  scale  definition  and  the  evidences  needed  for 
each  level,  towards  a  more  discipline  neutral  approach,  as 
the  TRL  assessment  is  being  consolidated  as 

interdisciplinary  and  relies  on  verification  and 
documentation  practices,  and  considering  that  IRL  scale 
is  evolving  from  a  data  integration  focus  to  a 
multidisciplinary  approach;  Explore  complementary 
methods  to  analyze  the  integration  readiness  through 
additional  systems  architecture  analysis  approaches,  given 
that  systems  architecture  is  a  vast  topic,  reflecting  the 
complexity  of  systems;  Analyze  a  system  with  multiple 
interface  types  between  its  elements  is  also  a  topic  not  yet 
explored  in  IRL  scale  literature. 
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This  paper  may  support  researchers  and  practitioners  to 
better  understand  the  IRL  scale  evolution,  opportunities  to 
the  scale  evaluation  process,  and  IRL  scale  relations  to 
the  system  architecture  and  integration  concepts. 

IRL  scale  complements  TRL  scale  as  a  tool  to  support 
decision  making  in  the  development  of  complex  systems. 
System  architects  need  to  decide  whether  and  how  to 
integrate  new  technologies  in  a  system  architecture,  being 
the  system  architecture  a  critical  factor  for  the  system 
success. 
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